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Overview 


A proposal entitled "A Distributed Analysis and Visualization System for Model and 
Observational Data" was jointly submitted by the University of Illinois and the NASA- 
Goddard Severe Storms Branch in response to NASA Headquarters’ Research 
Announcement, NRA-89-OSSA-21. Funding was provided to NCSA (National Center 
for Supercomputing Applications) and to NASA separately, with NASA funding ending 
early in the third year of NCSA funding. 

The work accomplished during the three year NASA funding was part of a larger effort to 
develop an analysis and display software system for handling and visualizing the massive 
amounts of data generated from satellites, observational field programs, and from model 
simulations during or after their execution. Components of the completed system include 
GEMPAK analysis, data handling, and display modules for which NASA had primary 
responsibility, and prototype software modules for three-dimensional interactive and 
distributed control and display, as well as data handling, for which NCSA was 
responsible. Overall process control was handled through a scientific and visualization 
application builder from Silicon Graphics known as the Iris Explorer. In addition, the 
GEMPAK related work (GEMVIS) was also ported to the Advanced Visualization 
System (AVS) application builder. Specifically, the NASA related objectives at NCSA 
included providing animation and three dimensional display, handling and displaying 
data from different sources and on different grids, and developing new analysis and 
visualization modules within the IRIS Explorer environment. 

This larger project within which these objectives were accomplished is known as 
PATHFINDER (Probing ATmospHeric Flows in an Interactive and Distributed 
EnviRonment). The mission of the PATHFINDER Project is to create a flexible, 
modular, and distributed environment for data handling, model simulations, data analysis, 
and presentation to be used in studying atmospheric and fluid flows, and which can be 
tailored for specific scientific research and weather forecasting needs. More specifically, 
in this environment the user should be able 

1) to manage large amounts (gigabytes and beyond) of data generated by satellites, 
observational field programs, and model simulations, 

2) to couple multiple and heterogeneous computers (e.g. SGI VGX, Cray Y-MP, 
CM-5), 

3) to utilize existing diagnostic and analysis software capabilities such as those 
found in GEMPAK (the GEneral Meteorological PAcKage built at Goddard 
Space Flight Center), 

4) to process multiple data streams(both model and observational) in creating a 
visualization, 

5) to access a variety of analysis and display capabilities including three dimensional 
rendering, animation, and collaborative tools for interacting with remote users on 
different workstations across the national network, and 

6) to incorporate video recording and playback, high definition monitors, and a 
virtual reality viewer. 
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PATHFINDER Modules Developed Under the NASA Award - General: 


NASA support enabled the development of many new SGI Explorer modules that are 
available freely to the NASA and national research community. The modules and 
documentation were developed in the context of existing commercial and public domain 
software packages such as SGI's Iris Explorer, GEMPAK, AVS, Inventor, HDF, netCDF, 
DTM and NCSA Mosaic. The final release of the NASA funded modules occurred on July 
18, 1994 and included 


1) Support for Explorer 2.2 (including both Inventor 1.1.2 and Openlventor 2.0) 

2) Complete online documentation (via WWW) 

http://redrock.ncsa. uiuc.eduyPATHFTNDER/pathrel2/top/top.html 

3) Full source code as well as ready to run binaries. 


Some highlights of this release are: 

HDF file support: Reading multiple SDS's from all existing versions of HDF files 
and reading and writing palettes and 24bit raster images 
NetCDF file support: Reading all data from Unidata or MFHDF3.3 netCDF tiles 
Visualization: Improved contour generation, three-dimensional vector display ectors, 

arbitrary 2D slicing, and particle advection 

Annotation and display: Adding materials, color bar annotation, 2D and 3D title 

New Grid datatype for Explorer: The Grid datatype is an extension to Lattice that 
allows metadata such as variable and coordinate names and units to be passed 
along with the data thru the Explorer map. A type extension file, a library, header 
files, some sample and utility modules are included. 

Other: Simple lattice math and filename generation for numbered sets of files 


PATHFINDER Modules Developed Under the NASA Award - Specific: 

The Iris Explorer modules developed under the NASA award were designed as part of the 
NSCA PATHFINDER project to aid in the analysis and visualization of atmospheric and 
other fluid flow data. Many of the modules were developed to fill in gaps in the 
capabilities provided by Explorer that were needed to accomplish NASA funded 
objectives. A more complete description follows. 


HDF file support 

The following modules provide support for the NCSA Hierarchical Data Format files 
(HDF). The HDF library (available on the NCSA ftp server, ftp.ncsa.uiuc.edu) provides a 
uniform method for storing multiple data fields in a single file, along with associated 
metadata. 


ReadDF: is capable of reading from a variety of HDF files (files created with HDF 
versions 3.1, 3.2, 3.3, MFHDF/3.3) and even Unidata netCDF files. The latter 
format translation is handled transparently. Up to five unique lattices may be 
generated from the file (as these files can contain multiple data fields) using a 
single module. A variety of dimensionalities and data types are supported also. 
Multiple variables may be combined in a single lattice to generate vector data. 
All three Explorer coordinate systems are supported. A user friendly interface 
provides control and information about the contents of arbitrary HDF and netCDF 
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files. (Note, HDF images and palette are not supported by this module, see the 
modules below) 

Read_H D F_Image : reads 24 bit raster images stored in HDF files (versions 3. 1-3. 2) 
and outputs them as a 2D lattice suitable for use with the Explorer image modules 
provided by SGI 

Read_HDF_P alette: reads palettes from HDF files (versions 3. 1-3.3) and outputs 
them in a format compatible with that of GenerateColormap. (1-D, 4 vector float 
lattice) (NCSA) 

Read_P alette: reads palettes from HDF files (versions 3. 1-3.3) or raw palettes and 
outputs them in a format compatible with that of GenerateColormap. (1-D, 4 
vector float lattice) A raw palette is a binary file containg 256 bytes of red values, 
followed by 256 bytes of green values, and 256 bytes of blue values. The module 
will automatically recognizes the file type. A single color palette can be outputted 
by selecting a color index from the current palette. 

Write _HDF_Image: writes 24 bit raster images to an HDF file (version 3.3) input is a 
2D lattice. 

Write, _HDF JP alette: writes palette from GenerateColormap to an HDF file (version 
3.3) as an HDF palette. 

Visualization 

Contour2: an improved version of the Contour module that comes with Explorer. 
Allows the specification of either the number of contour levels desired or a 
specific contour interval. The data value range that is contoured can be either 
fixed or data dependent. The zero contour line can be hidden and negative 
contour lines can be displayed differently to distinguish them if desired. 

ParticleAdvect: generates points (random, uniform or ring structure) in a user 

specified subvolume of a given vector lattice field and advects all the points until 
they are out of the data domain. The user specifies step size and has choice for 
using adaptive/fixed step size, integration method (Euler, second-order Runge- 
Kutta and fourth-order Runge-Kutta) and interpolation method (trilinear/nearest 
neighbor). 

Streakline2: a slightly modified version of the Explorer Streakline module. The 
main change is that the controls to manually specify a release point from the 
control panel without having to use picking have been enabled. (Only .mres and 
.help files are provided as it uses the existing Streakline binary) 

VectorDisp: a much enhanced version of the Vector module that comes with 

Explorer New features include several vector styles, adjustable vector length, 
arrowhead size and scaling (including an automatic scaling mode), and a variety 
of methods to map colors to the vectors. 

Vslice: takes an arbitrary 2D vertical slice from a 3D lattice. The location of the slice 
is specified by selecting two end points of the slice in a small window on the 
control panel. A line is drawn between the two selected points and extended to the 
boundaries. Contour lines of a horizontal plane are drawn as a reference. Unlike 
OrthoSlice, the slice does not need to be orthoginal. 
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AddMaterial: adds a material to a geometry stream in an Explorer map. Sliders and a 
material editor widget allow easy specification of material properties. Unlike 
materials set from inside Render which must be reset with each use, the values are 
saved as module parameters between sessions. 


ColorKey: takes a color map and generates a horizontal colored bar that displays the 
gamut of colors and tick marks with labels indicating the correspondence between 
colors and data values. This is sent to Render to annotate a visualization. 


Display: an enhanced version of Render. The main addition is the ability to capture 
new frames as they are rendered (as in a map that contains a loop) and then play 
back the captured frames at various speeds. The actual geometried are stored 
instead of the images. 

Text2D: generates a simple 2D title as a geometry to be sent to annotation port on 
Render. The position, size and font are set from the control panel. 

Text3D: like Text2D, except that 3D text is used. This allows it to be included 
within the visualization as normal geometries. 3D text is always oriented 
correctly for reading no matter how the object is rotated in Render. The 
components of the 3D font, front back and sides, may be turned on or off 
individually. (NASA) 

WireFrame2: an enhanced version of WireFrame that allows control over the 

wireframe's material. 



These very simple modules are written in the Explorer Shape language and can serve as 
examples for your own lattice math modules. The SubtractLat module is particularly 
useful for finding the difference between two lattices that are being compared. 


AddLat: Adds two lattices together, point by point _ 

DivideLat: Divides one lattice by another, point by point 

MultiplyLat: Multiplies one lattice by another, point by point 

SubtractLat: Subtracts one lattice from another, point by point 


Miscellaneous 

TimeFilename: This module will construct file names from 4 components: path(string), 
name(string), time(numerical value) and extension (string). A printf() like format string 
specifies how these components are to be combined and formatted. The resulting 
filename may be passed as a parameter to modules that read to or write from files. It is 
useful to allow a loop in a map to access a series of sequentially numbered files. 
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In addition to the predefined data types that come with Explorer such as lattice, 
parameter, pyramid, etc., SGI also provides a mechanism to create custom data types. 
This is different from the "Unknown" data type. These new types are dynamically loaded 
into Explorer at startup time and become as much a part of Explorer as the standard ones. 
For example, they show up in port names and in the module builder as valid choices for 
port types. 

The Grid data type is essentially an extention of the Explorer lattice. In fact, it contains a 
lattice within its structure. In addition to the data carried within the lattice definition, it 
provides an arbitrary name space for additional metadata, such as variable and dimension 
names, units, etc. Some of these new metadata fields are predefined, but any arbitrary 
piece of metadata can be included. 

The Grid’s namespace works very much like the environment of a C shell (csh). It 
consists of a series of keyword, string value pairs. An example would be: UNITS=g/kg 
where UNITS is the keyword and the string "g/kg" is the value. 

Data file formats such as HDF, netCDF and GEMPAK grid files often contain useful 
metadata that describes the actual data. A limited amount of this information such as 
dimension sizes and coordinates may be passed from module to module using the 
Explorer lattice type. Other information such as variable and dimension names, units, 
missing value flags, etc. is left behind in the file. The Grid type provides space to carry 
this information from module to module. User developed modules can make use of this 
information for things like automatic geometry annotation and missing value handling. 


Utilities and Sample Modules for the Grid Data Type 
Converters: 

LatToGrid: converts a lattice to a Grid. Additional Grid information 
is supplied via type-in boxes on the control panel. 

GridToLat: converts a Grid back into a lattice 
Information: 

PrintGridlnfo: prints out the contents of the Grid namespace. 

Sample Visualization modules using Grid type: 

Contour2G: same as Contour2 (in General directory) except uses Grids 
instead of lattices. It handles missing values. 

GridTitle: generates 2 and 3D text for annotation. This text can contain 
references to Grid keywords for which the module automatically 
substitutes the corresponding values from an input Grid. Is useful for 
automatically annotating a display based on input data. Multiline text is 
supported. The module has controls for text font, scaling, placement and 
other properties. 

OrthoSlice2G: same as OrthoSlice but uses Grids 
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ReadDFG: same as ReadDF (in General directory) except outputs Grids 
instead of lattices. Some of the additional metadata in the data files is 
used to fill in the Grid fields. 

WireFrame2G: same as PATHFINDER WireFrame2 but accepts a Grid input. 
The Grid Library 

The Grid Library is provided for developing your own Grid modules. It includes a 
comprehensive API designed after Explorer's lattice access API so that it will be easy for 
Explorer programers to use. 


GEMVIS 

GEMVIS is software for the interactive analysis and three-dimensional visualization of 
atmospheric data. It is based on GEMPAK (The General Meteorological Package); it 
reads GEMPAK grid files and includes various GEMPAK analysis and map projection 
capabilities in the form of modules. This version runs within the Iris Explorer or AVS 
environments and enhances GEMPAK capabilities with three dimensional visual 
displays. GEMVIS was developed by NASA / Goddard Space Flight Center and NCSA 
as part of the NCSA Pathfinder project. GEMVIS is provided in source code format only 
due to restrictions on distributing pre-compiled versions containing the GEMPAK 
libraries. 

To build GEMVIS it is necessary to link to GEMPAK subroutine libraries. These 
libraries are created as part of the standard GEMPAK build procedure. GEMP AK can be 
obtained from the Unidata Program Center, a division of the University Corporation for 
Atmospheric Research ((303) 497-8678 or support@unidata.ucar.edu) or from COSMIC 
(the NASA Computer Software Management and Information Center, (706) 542-3265 or 
service@cossack.cosmic.uga.edu). 

The set of capabilities targeted to the GEMPAK user community includes: 

GEMPAK grid diagnostics & file access: 

the full range of GEMPAK functions and operators supported 

a new operator implemented for 3D vector data 

all GEMPAK grid data formats supported 

all vertical coordinate systems handled 

relevant modules: GemGetGrid , GemGridFunction 

Map projections & topography : 

a wide variety of map projections 
maps registered with data set visualizations 
topography displayed in appropriate projections 
relevant modules: GemDrawMap, GemTopo, GemGetGrid, and 
GemGridMapping 

A visualization obtained using GEMVIS is shown in the following figure. The lower left 
window contains a potential vorticity surface over a topographical map of the eastern part 
of the U.S. along with a vertical slice of the three-dimensional winds. The slice is taken 
through a line the user forms by selecting two points over a map of the U.S. as shown in 
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the upper left window. The upper right window contains the Iris Explorer map used in 
creating the three dimensional image. It contains some of modules written as part of the 
PATHFINDER project funded by NASA. 



Technology Transfer 

The Numerical Algorithms Group of the U.K. provides Explorer support for SGI and is 
working to port Explorer to other platforms, selected PATHFINDER modules we have 
developed will be included in their work. 

Other Related PATHFINDER Activities 

More research oriented accomplishments in the context of PATHFINDER include the 
prototyping of a distributed computer simulation control system that interfaced a 
computer model running on a supercomputer with the Explorer environment and the 
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prototyping of an Inventor based 3-D animation tool as well. In addition, beta testing of 
new SGI Explorer versions was carried out. An visualization of these other activities is 
shown on the next page. A growing severe storm is shown together with the associated 
SGI Explorer dataflow network (including some of the modules developed as part of the 
PATHFINDER project funded by NASA). An interactive window for controlling the 
storm simulation on a Cray Y-MP is shown in the lower left. In the lower right the cloud 
surfaces and horizontal contours are automatically updated as the simulation proceeds. 
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